Add new default date format #5163

kolchfa-aws · 2023-10-05T14:55:56Z

Fixes #5158

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Fanit Kolchina <[email protected]>

kolchfa-aws · 2023-10-05T14:56:29Z

@CaptainDredge Could you review this PR for technical accuracy?

CaptainDredge · 2023-10-06T07:10:23Z

_field-types/supported-field-types/date.md

+A date math expression contains a fixed date, optionally followed by one or more mathematical expressions. The fixed date may be either `now` (current date and time in milliseconds since the epoch) or a string ending with `||` that specifies a date (for example, `2022-05-18||`). The date must be in the `strict_date_time_no_millis||strict_date_optional_time||epoch_millis` format.
+
+If you specify multiple date formats in the field mapping, OpenSearch uses the first format to convert the milliseconds since the epoch value to a string. <br>
+If a field mapping for a field contains no format, OpenSearch uses `strict_date_optional_time` to convert the epoch value to a string.


Suggested change

If a field mapping for a field contains no format, OpenSearch uses `strict_date_optional_time` to convert the epoch value to a string.

If a field mapping for a field contains no format, OpenSearch uses `strict_date_optional_time` format to convert the epoch value to a string.

CaptainDredge · 2023-10-06T10:30:30Z

@reta should we add here that user can revert to older format by disabling feature flag, would like to know your opinion

reta · 2023-10-06T13:14:09Z

@reta should we add here that user can revert to older format by disabling feature flag, would like to know your opinion

@CaptainDredge Yes please, we have a new feature flag that should be documented, thank you. The 2.11 was cut, unless we get the green light to backport it to 2.11, this would be 2.12 feature.

_field-types/supported-field-types/date.md

Signed-off-by: kolchfa-aws <[email protected]>

kolchfa-aws · 2023-10-06T13:38:29Z

@CaptainDredge I have reworded to incorporate your suggestion. Please let me know:

what's the feature flag, how the user can enable the feature flag, and what's the behavior when it's enabled/disabled
if this is still going into 2.11
Thanks

reta · 2023-10-06T13:41:44Z

what's the feature flag, how the user can enable the feature flag, and what's the behavior when it's enabled/disabled

Just an example of feature flags in the system (and how we document them)

CaptainDredge · 2023-10-06T14:49:46Z

what's the feature flag

opensearch.experimental.optimization.datetime_formatter_caching.enabled

how the user can enable the feature flag

Note that the above feature flag is enabled by default, user has to disable it if he/she wants to go to the previous default formatter

Disabling the feature flag

There are several methods for disabling the feature flag, depending on the installation type.

Disable on a node using a tarball installation

The flag is toggled using a new jvm parameter that is set either in OPENSEARCH_JAVA_OPTS or in config/jvm.options:

Option 1: Update config/jvm.options by adding the following line:

-Dopensearch.experimental.optimization.datetime_formatter_caching.enabled=false

Option 2: Use the OPENSEARCH_JAVA_OPTS environment variable:

export OPENSEARCH_JAVA_OPTS="-Dopensearch.experimental.optimization.datetime_formatter_caching.enabled=false"

Disable with Docker containers

If you're running Docker, add the following line to docker-compose.yml underneath the opensearch-node and environment sections:

OPENSEARCH_JAVA_OPTS="-Dopensearch.experimental.optimization.datetime_formatter_caching.enabled=false"

what's the behavior when it's enabled/disabled

Its enabled by default and therefore opensearch will use the new default format introduced along with an optimisation which cache last successfully parsed formatter to parse the datetime field for the next document getting indexed in the same shard. When the featureflag is disabled opensearch will use the old format along with disabling the above optimisation

what's the behavior when it's enabled/disabled

I couldn't get the green light for the backport PRs to go in 2.11. I think it'll now go in 2.12 😞

CaptainDredge · 2023-10-06T14:52:25Z

@reta shouldn't we ideally have a common section for instructions on enabling and disabling feature flag instead of mentioning the process in every feature. This will allow us to just then document the new feature flag briefly with a relative link to the featureflag enable/disable documentation

reta · 2023-10-06T14:54:56Z

@reta shouldn't we ideally have a common section for instructions on enabling and disabling feature flag instead of mentioning the process in every feature

Totally, we could ask documentation team to help us with restructuring the docs to have common details on feature flags and than we could only mention the flag name(s) in the feature specific docs, @kolchfa-aws how does it sound? thank you!

kolchfa-aws · 2023-10-06T14:58:14Z

@reta Are all feature flags enabled in the same way? If so, we definitely need to put it in one place.

CaptainDredge · 2023-10-06T15:00:11Z

@kolchfa-aws Yes all the feature flags are enabled in the same way

reta · 2023-10-06T15:15:15Z

@kolchfa-aws Yes all the feature flags are enabled in the same way

All the same way, correct.

CaptainDredge · 2023-10-20T12:53:51Z

@kolchfa-aws this change has been backported to 2.x and will be released in v2.12. What's the way ahead to get this documentation merged now?
opensearch-project/OpenSearch#10448

natebower

@kolchfa-aws LGTM!

Signed-off-by: Fanit Kolchina <[email protected]>

…ocumentation-website into date-update

kolchfa-aws · 2023-10-23T22:42:59Z

@CaptainDredge @reta I added a "how to enable experimental features" section. Could you review this commit to make sure everything looks good? 03edad8. Then we can merge.

CaptainDredge

LGTM 🚀

natebower

@kolchfa-aws Just one comment. Thanks!

natebower · 2023-10-25T14:17:50Z

_field-types/supported-field-types/date.md

@@ -223,7 +227,7 @@ GET testindex/_search

 The date field type supports using date math to specify durations in queries. For example, the `gt`, `gte`, `lt`, and `lte` parameters in [range queries]({{site.url}}{{site.baseurl}}/query-dsl/term/range/) and the `from` and `to` parameters in [date range aggregations]({{site.url}}{{site.baseurl}}/query-dsl/aggregations/bucket/date-range/) accept date math expressions.

-A date math expression contains a fixed date, optionally followed by one or more mathematical expressions. The fixed date may be either `now` (current date and time in milliseconds since the epoch) or a string ending with `||` that specifies a date (for example, `2022-05-18||`). The date must be in the `strict_date_time_no_millis||strict_date_optional_time||epoch_millis` format.
+A date math expression contains a fixed date, optionally followed by one or more mathematical expressions. The fixed date may be either `now` (current date and time in milliseconds since the epoch) or a string ending with `||` that specifies a date (for example, `2022-05-18||`). The date must be in the [default format](#default-format), which by default is `strict_date_time_no_millis||strict_date_optional_time||epoch_millis`.


The last sentence might be a bit redundant (the default format by default is).

Yes, but there are two options for the default format and one of them is the default one :)

Signed-off-by: Fanit Kolchina <[email protected]>

* Add new default date format Signed-off-by: Fanit Kolchina <[email protected]> * Update _field-types/supported-field-types/date.md Signed-off-by: kolchfa-aws <[email protected]> * Add enabling experimental features section Signed-off-by: Fanit Kolchina <[email protected]> --------- Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: kolchfa-aws <[email protected]>

Add new default date format

995822e

Signed-off-by: Fanit Kolchina <[email protected]>

kolchfa-aws added release-notes PR: Include this PR in the automated release notes v2.11.0 labels Oct 5, 2023

kolchfa-aws requested a review from hdhalter as a code owner October 5, 2023 14:55

kolchfa-aws self-assigned this Oct 5, 2023

kolchfa-aws requested review from Naarcha-AWS, vagimeli, ananzh, seanneumann, AMoo-Miki and natebower as code owners October 5, 2023 14:55

CaptainDredge mentioned this pull request Oct 6, 2023

[Backport 2.x] Performance Improvement for Datetime formats to 2.x opensearch-project/OpenSearch#10448

Merged

7 tasks

CaptainDredge suggested changes Oct 6, 2023

View reviewed changes

CaptainDredge mentioned this pull request Oct 6, 2023

[Backport 2.11] Performance Improvement for Datetime formats to 2.11 opensearch-project/OpenSearch#10453

Closed

7 tasks

kolchfa-aws commented Oct 6, 2023

View reviewed changes

_field-types/supported-field-types/date.md Outdated Show resolved Hide resolved

Update _field-types/supported-field-types/date.md

bda13f2

Signed-off-by: kolchfa-aws <[email protected]>

kolchfa-aws added v2.12.0 and removed release-notes PR: Include this PR in the automated release notes v2.11.0 labels Oct 6, 2023

kolchfa-aws modified the milestone: v2.12 Oct 6, 2023

hdhalter added this to the v2.12 milestone Oct 6, 2023

natebower approved these changes Oct 20, 2023

View reviewed changes

kolchfa-aws added the release-notes PR: Include this PR in the automated release notes label Oct 20, 2023

kolchfa-aws added 2 commits October 23, 2023 18:40

Add enabling experimental features section

03edad8

Signed-off-by: Fanit Kolchina <[email protected]>

Merge branch 'date-update' of https://github.com/opensearch-project/d…

3d7d3f6

…ocumentation-website into date-update

CaptainDredge approved these changes Oct 25, 2023

View reviewed changes

natebower approved these changes Oct 25, 2023

View reviewed changes

Reworded the default sentence

c07e784

Signed-off-by: Fanit Kolchina <[email protected]>

kolchfa-aws merged commit 49889b4 into main Oct 25, 2023
3 checks passed

Naarcha-AWS deleted the date-update branch March 28, 2024 23:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add new default date format #5163

Add new default date format #5163

kolchfa-aws commented Oct 5, 2023

kolchfa-aws commented Oct 5, 2023

CaptainDredge Oct 6, 2023

CaptainDredge commented Oct 6, 2023

reta commented Oct 6, 2023 •

edited

Loading

kolchfa-aws commented Oct 6, 2023

reta commented Oct 6, 2023

CaptainDredge commented Oct 6, 2023 •

edited

Loading

CaptainDredge commented Oct 6, 2023

reta commented Oct 6, 2023

kolchfa-aws commented Oct 6, 2023

CaptainDredge commented Oct 6, 2023

reta commented Oct 6, 2023

CaptainDredge commented Oct 20, 2023

natebower left a comment

kolchfa-aws commented Oct 23, 2023

CaptainDredge left a comment

natebower left a comment

natebower Oct 25, 2023

kolchfa-aws Oct 25, 2023

	If a field mapping for a field contains no format, OpenSearch uses `strict_date_optional_time` to convert the epoch value to a string.
	If a field mapping for a field contains no format, OpenSearch uses `strict_date_optional_time` format to convert the epoch value to a string.

Add new default date format #5163

Add new default date format #5163

Conversation

kolchfa-aws commented Oct 5, 2023

Checklist

kolchfa-aws commented Oct 5, 2023

CaptainDredge Oct 6, 2023

Choose a reason for hiding this comment

CaptainDredge commented Oct 6, 2023

reta commented Oct 6, 2023 • edited Loading

kolchfa-aws commented Oct 6, 2023

reta commented Oct 6, 2023

CaptainDredge commented Oct 6, 2023 • edited Loading

Disabling the feature flag

Disable on a node using a tarball installation

Disable with Docker containers

CaptainDredge commented Oct 6, 2023

reta commented Oct 6, 2023

kolchfa-aws commented Oct 6, 2023

CaptainDredge commented Oct 6, 2023

reta commented Oct 6, 2023

CaptainDredge commented Oct 20, 2023

natebower left a comment

Choose a reason for hiding this comment

kolchfa-aws commented Oct 23, 2023

CaptainDredge left a comment

Choose a reason for hiding this comment

natebower left a comment

Choose a reason for hiding this comment

natebower Oct 25, 2023

Choose a reason for hiding this comment

kolchfa-aws Oct 25, 2023

Choose a reason for hiding this comment

reta commented Oct 6, 2023 •

edited

Loading

CaptainDredge commented Oct 6, 2023 •

edited

Loading